Multi-Threaded Dynamic Per-File Read-Ahead Cache for Deduplication System

ABSTRACT

A multi-stream restore method for reducing restore time of file data to service an external read request for file data stored by a data deduplication system employs a per-file read-ahead cache that is populated by multiple parallel internal read-ahead streams. The number of parallel read-ahead streams is dynamically variable based upon processing conditions and a set of heuristically pre-determined criteria that characterize the file data including a data access pattern that is sequential and a processing load that is below a predetermined threshold. Processing conditions are periodically monitored and adjustments to the multi-stream data access are made to optimize performance.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.16/389,260, filed Apr. 19, 2019, the disclosure of which is incorporatedby reference herein.

BACKGROUND

This invention relates generally to accessing data files from large datadeduplication filesystems, as for restore operations, and moreparticularly to a system and method for multi-threaded read-ahead dataaccess to increase access speed and reduce input/out (“I/O”) costs.

In many filesystems, sequential data restores are serialized. One way toincrease access speed and amortize overhead costs of I/Os in thesefilesystems is to anticipate sequential data reads and issue prefetchrequests (also called “read-aheads”) for the anticipated data. Suchprefetch requests are treated as hints, and the data corresponding tothese read requests are stored in buffers, referred to as a “read-aheadcaches”, from which the next sequential data reads are serviced.

In large backup storage filesystems, new generations of backups arewritten frequently. Thus, over time the locality of file datadeteriorates in deduplication environments. The serialization of readscompounds the effects of bad locality and causes further degradation inrestore performance. The reads on data deduplication systems aretypically few and far between. They are normally to restore the backups,and most of the time, the reads are sequential reads. Restores for fileswith bad locality are slow because of the serialization and the need formultiple I/Os to service a request. This makes restores moretime-consuming, and resource and I/O costs are greater.

It is desirable to provide systems and methods that avoid the foregoingand other problems associated with file restores from large data storagefilesystems by reducing the time and I/O overhead required for restoringfiles so that restores become faster and more efficient.

The invention addresses these and other associated problems associatedwith data restores by affording a system and method that dynamicallychanges file accesses such as the number of read-ahead threads basedupon predetermined rules that relate to, among other things, prevailingconditions within the filesystem and the data file being accessed sothat file accesses and restores are optimized.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagrammatic view of an exemplary system of the type inwhich the invention may be used;

FIGS. 2A and 2B illustrate two different patterns of data access from adata deduplication filesystem, FIG. 2A illustrating a single data streamaccess, and FIG. 2B illustrating a multi-stream access;

FIG. 3 is a diagrammatic view of a multi-stream read-ahead cache inaccordance with the invention;

FIG. 4 is a diagrammatic view of a process in accordance with theinvention for reading data in a data chunk unit;

FIG. 5 is a diagrammatic view illustrating a multi-stream process inaccordance with the invention for reading a data file in a filesystem;

FIG. 6 is a diagrammatic view of a process in accordance with theinvention for determining whether a filesystem data access process willbe single stream or multi-stream;

FIG. 7 is a diagrammatic view illustrating a portion of the process ofFIG. 6 in more detail; and

FIG. 8 is a process in accordance with the invention for periodicallymonitoring a filesystem to determine whether to service filesystem readsusing a multi-stream or single stream process.

DESCRIPTION OF PREFERRED EMBODIMENTS

The invention is particularly advantageous in accessing data files in adata deduplication filesystem, as for a restore operation, and will bedescribed in that context. However, as will become evident from thefollowing, this is illustrative of only one utility of the invention,and it will be appreciated that the invention has broader utility andthat it may be used to advantage for data access in other types offilesystems and for other types of applications.

As will be described in more detail herein, in one aspect the inventionoptimizes access to file data by reducing access times by servicingexternal read requests for file data using a per-file read-ahead cachethat may be continuously enabled and populated by multiple parallelrestores when the system has sufficient resources and capability. If thedata access pattern is sequential and the system has the capability,corresponding multiple read-aheads may be issued. To service an externalread request for data, the read-ahead cache may be populated by adynamically varying number of multiple internal read-ahead threads thatcorrespond to the one external read stream. The number of read-aheadsissued depends upon certain prevailing system processing conditions andpredetermined heuristically determined criteria or rules (referred toalso herein as heuristics). To afford parallelism for the internalreads, the read-ahead cache is operated at a greater offset than acurrent read offset, and the internal reads may be distributed acrossthe multiple streams. The read-ahead threads work in parallel onportions of a file referred to as chunks, and the number of threads maybe dynamically changed, e.g., threads may be turned on and off, asconditions change, as will be described. This enables data accesses andI/Os to be optimized, while giving a multi-streamed advantage to certaindata reads. This approach is referred to herein as a multi-streamrestore (“MSR”).

FIG. 1 is a high-level functional block diagram of a system of the typein which the invention may be practiced. The system may be part of anetworked data deduplication storage system, such as is available fromDataDomain, a subsidiary of the assignee of the present invention. Thesystem may comprise a purpose built backup appliance (“PBBA”) node 10that is interfaced via a network 12 to a plurality of servers or clients14 running different applications. Node 10 may comprise a processorwhich serves as a data deduplication (“DDUP”) engine and a memory 16.The memory may store computer executable instructions which control theprocessor to perform the functions and operations described herein, andmay additionally afford in-memory data structures that serve asread-ahead caches for servicing different read requests for file data.

The node may additionally comprise a storage layer 18, e.g., comprisinga plurality of data storage devices such as disks that serve as acontent store, a data deduplication filesystem (“DDFS”) 20 that providesfilesystem services which communicate with the storage layer, and othermanagement services 22. The DDFS PBBA file system is designed primarilyfor writes and to have a few very large files.

There are typically few restores on a DDFS filesystem. The restores areprincipally to restore backups, and they are normally sequential. Therequests (in PBBA restores) are serialized. Additionally, filesystemdevelopers frequently try to tune the filesystem for maximum throughputat a given number of concurrent streams. If the number of streams islower than the maximum throughput number of streams the performance isusually sub-optimal. Restores, as mentioned above, are few and farbetween. But when they happen, they are important actions on the system.Serial execution of read requests, together with a fixed prefetchhorizon makes sub-optimal use of the I/O bandwidth.

FIGS. 2A and 2B illustrate application reads that access data in thestorage layers 18 in response to an external read request 26 for filedata from an application. FIG. 2A shows a single stream internal readthread 28 responding to the read request 26, and FIG. 2B shows multipleinternal read threads (Steams 1-4) in a multiple stream restore (MSR)operation responding to a read request 26. As will be described, toservice a read request the invention may dynamically change from asingle internal read thread to multiple read threads in order tooptimize use of available system resources.

However, by being generically sized for maximum throughput, the systemis still serialized by the requests and the locality on disk. Theinvention addresses this by affording parallel restores when the systemhas sufficient resources. The external reads are serviced by theread-ahead per file cache, as shown in FIG. 3.

FIG. 3 illustrates diagrammatically a MSR operation in which multipleinternal read streams (threads) 30 access data in a deduplication engine32 to populate a read-ahead cache 34 to service an external applicationread request 36. Multiple internal read streams may be openedcorresponding to the external read request, and the internal reads maybe distributed across these multiple internal read streams. Aspreviously described, and as will be described further, the number ofstreams to populate the read-ahead cache may vary dynamically dependingupon the prevailing operating conditions and heuristicallypre-determined rules and criteria that control the number of readstreams enabled. The criteria, as will be described, are designed tooptimize data access time to afford quick restores, and to minimize theI/O overhead costs.

The read ahead cache 34 of FIG. 3 being filled in by the multipleinternal read streams 30 affords the desired parallelism. Each internalread thread preferably reads a data increment within a section of thefile which is referred to as a chunk, where increments and chunks havepreselected ranges (sizes). As an illustration, FIG. 4 shows an exampleof a chunk 40 having a range of 32 MB, and a read increment 42 (referredto as a prefetch increment) having a granularity size of 1 MB. In thefigure, the read increment may begin at a point within the chunk at aprefetch horizon 44 (a parameter of the filesystem), as e.g., at 24 MB,measured within the chunk from the end of a read offset 48 that ismeasured from a starting position 50 of the chunk. The prefetch horizonspecifies how far from the read offset the prefetch increment begins.Once an internal read thread has completed reading its chunk, theinternal read thread may move to a next designated chunk, which may notbe the next contiguous section of the file to be read. Rather, the nextdesignated chunk may be determined as:

next chunk offset=current chunk start offset+(number of internalthreads*chunk size)

This is illustrated by the following example.

Referring to FIG. 5, assume that there are four read streams, S0-S3, andthe chunk size is 32 MB as shown in FIG. 4. Assume further that thefirst stream SO reads a first chunk C0 (0-32 MB). The second stream S1reads the next second chunk C1 (32-64 MB); the third stream S2 reads thenext third chunk C2 (64-96 MB); and the fourth stream S3 reads the nextfourth chunk C3 (96-128 MB). After stream SO completes reading C0, S0then reads the next chunk C4 (128-160 MB) to be read, as indicated bythe dotted line. Similarly, stream S1 next reads C5; S2 next reads C6;and S3 next reads C7, as shown.

In accordance with the invention, the RA cache for multi-stream read(MSR) operation is preferably not automatically enabled, i.e., MSR isnot enabled upon an external read request first being received. Rather,MSR is preferably enabled based upon the prevailing operating(processing) conditions at that particular time and upon a set ofpredetermined rules (criteria). An example of a rule for enabling MSRmay be if the reads are sequential and if the file size is greater thana predetermined minimum size, the RA cache may be enabled for MSR. Anon-exclusive set of criteria that the invention may employ singularlyor in combinations for enabling and/or controlling MSR during operationincludes the following:

-   -   Sequential access pattern: MSR is enabled if the read access        pattern is sequential;    -   File size: MSR is enabled if the file size is greater than a        predetermined threshold (e.g., 10 GB) because setting up MSR is        time consuming and it is preferably avoided unless the        performance benefit outweighs the costs;    -   Tier of storage: the I/O bandwidth for certain storage tiers may        be too small to waste CPU resources on MSR, so MSR is not        enabled for reads from these tiers;    -   Read Offset: a minimum read offset (e.g., 100 MB) may be set for        enabling MSR; sequential read requests; and to eliminate false        positives;    -   Number of streams: the system may be sized for processing a        maximum number of streams; an additional stream may exceed this        number and slow read access;    -   Back-off mechanism and time check: to avoid the costs associated        with testing all rules on each read request; upon a test failing        for a stream, a back-off factor n (e.g., an incrementing        integer) may be set to establish a retry time before a retry is        permitted in n*t seconds(s) (e.g., t=10 s); the first time a        test fails, the back-off factor may be set to one, and retry may        be set to 10 s; the second back-off factor may be two and retry        is 20 s, etc. This amortizes the cost of heuristic checks to        near zero;    -   System load: the RA cache and MSR may not be activated on a        stream if the system processing load exceeds a predetermined        threshold.

In addition, the RA cache may monitor system processing load and it maydisable itself if the system load increases beyond the predeterminedthreshold. Otherwise, continued operation may have deleterious effectson other performance of the system, such as ingest or replication. Thismay be achieved, in one embodiment, by monitoring system processingoperations, as, for instance, by polling the system load periodically,or by registering call-backs into a quality of service (QoS) subsystem.Some characteristics that may be monitored include, for example, thenumber of active reads and writes (streams) on the system, CPU usage, ordisk or I/O responses, among others, that occur within a preselectedperiod of time.

FIGS. 6-8 are functional flow diagrams that illustrate processes inaccordance with the invention for enabling, monitoring and controllingMSR operations in a data storage system such as shown in FIG. 1.

FIG. 6 is a diagrammatic view of a process in accordance with theinvention for determining whether a filesystem data access process willbe single stream or multi-stream in responding to an external readrequest. The process may be performed by a processor of the PBBA, forexample, executing computer readable instructions stored in theassociated memory.

Referring to FIG. 6, at 60 a request for a read may be received from anapplication. At 62, it is determined whether MSR is already enabled. Ifit is, the received read request is queued at 64 for execution as amulti-stream restore, and the initial determination ends at 66. Ifinstead at 62 MSR is not enabled, at 68 the process determines whetherMSR is eligible to be enabled as by determining the system operatingconditions and by applying rules such as described above, as will bedescribed below in connection with FIG. 7. If, at 68, it is determinedthat MSR is eligible to be enabled, at 70 MSR is activated and thenreceived read request is serviced as a MSR using multiple internal readstreams to populate the RA cache with file data, from which the filedata is returned to the requesting application. If instead at 68 it isdetermined that MSR is not eligible, e.g., because of current systemload, the read request is serviced at 72 without MSR as a single stream.

When read requests begin arriving, they may be initially serviced bysingle stream reads. FIG. 7 illustrates a process for determiningwhether to establish MSR when a read arrives. It also illustrates thestep 68 of the process of FIG. 6 that determines whether MSR is eligibleto service the received read request.

In the embodiment of the process shown in FIG. 7, a number of criteriamust be satisfied in order to activate MSR. As shown, at 74 a firstcriterion that must be satisfied may be whether the file size is greaterthan a predetermined minimum file size. As explained above, setting upMSR is costly and time consuming. Unless the file size is greater than aminimum size, such as 10 GB, the increased performance benefits of MSRmay not justify the cost and time of MSR for file sizes that are lessthan the predetermined minimum size. If the file size meets thiscriterion, the process proceeds to step 76. If not, the process exits tostep 78 without setting MSR and services the read request using a singlestream read of the content store. It also sets at 78 a predeterminedbackoff factor and time, as described above, to re-execute the loop ofFIG. 7 to recheck for eligibility of MSR.

At step 76, the process determines whether the second criteria issatisfied, for instance one that requires that the file offset begreater than a predetermined minimum file offset, e.g., 100 MB. Thispermits looking back to determine the file access pattern in order todetermine whether to enable MSR. If the file offset criterion is notsatisfied, the process proceeds to step 78 as described above. If thefile offset criterion is satisfied, the process continues to step 80where it is determined if the file read access pattern is sequential inorder to activate MSR. If the access pattern is not sequential, theprocess exits to step 78. If this file access criterion is satisfied atstep 80, the process proceeds to step 82 where the fourth criteriondetermines whether the system is not loaded above a predeterminedthreshold. If it is not so loaded, the process proceeds to step 84 andactivates MSR. Otherwise, the process exits to step 78 to set a MSRbackoff for retry.

The set of criteria used in the process of FIG. 7 for deciding whetherto enable MSR have been determined heuristically to be useful, and theyare exemplary. Other different criteria and different numbers ofcriteria may be employed in addition or in substitution to those shown,such as, for example, the storage tier of the file data requested, andthe number of streams in use relative to the maximum permissible numberbased upon the system capabilities. Moreover, in an embodiment, theprocess of FIG. 7 may be repeatedly executed at predetermined timeintervals to check for MSR eligibility based upon the system load andother conditions such as the parameters of the file that is the subjectof the external read request in order to optimize the processingcapability of the system. This is illustrated in FIG. 8.

FIG. 8 shows a flow diagram of a process for periodically re-executingthe MSR read service process to monitor whether the system load exceedsa predetermined threshold to continue or exit MSR.

Referring to FIG. 8 at 90 the process begins the MSR service routineloop process. At 92, a check is made to determine whether the time sincea last load check exceeds a predetermined threshold. If the timethreshold does not exceed the threshold, the process proceeds to step 94where the external read request is serviced from the cache. On the otherhand, if at 92 the time does exceed the time threshold, the system loadis checked at 98 and at 100 it is determined whether the system loadexceeds a predetermined threshold load. If it does not, the processproceeds to step 94 where the read request is serviced from the cache.If the system load does exceed the threshold, at 102 the process exits(teardowns) the MSR loop and reverts to a single stream read.

The processes of FIGS. 7 and 8 may be modified to incorporate steps thatdynamically adjust the number of multi-stream reads in real time basedupon the system load and other operating conditions such as theparameters of the files being read.

While the foregoing description has been with reference to particularembodiments, it will be appreciated by those skilled in the art thatchanges from these embodiments may be made without departing from theprinciples of the invention, the scope of which is defined by theappended claims.

1. A method of improving access time of sequential data in a file in astorage layer of a data system to service an external read request for adata restore, comprising: opening multiple internal read-ahead streamsthat read ahead in parallel data in increments of a chunk of said filein said storage layer to prefetch said data; dynamically varying, tooptimize processing and speed of said data access, the number of saidmultiple read-ahead streams that read said data or a prefetch horizon ofthe data read by each said read-ahead stream based upon the presentsystem processing conditions and based upon a set of preselectedcriteria applicable to characteristics of the file being read;populating a cache with the data read by said multiple read-aheadstreams; and servicing said external read request with the datapopulated in the cache.
 2. The method of claim 1 further comprisingconfirming, prior to opening said multiple read-ahead streams that thesize of said file being accessed is greater than a predeterminedthreshold size below which size the costs of setting up said multipleread-ahead stream access exceed the performance benefit obtained frommultiple read-ahead stream access.
 3. The method of claim 1, whereinsaid preselected criteria include said file data being stored in astorage tier having an input/output (I/O) bandwidth that is smaller thanone at which the processing resource costs required for multiple streamreading exceed the performance benefit obtained.
 4. The method of claim1, wherein said present system processing conditions comprise a systemprocessing load below a predetermined threshold load corresponding tooptimum input/output (I/O) processing.
 5. Then method of claim 1,wherein said dynamically varying the number of multiple read-aheadstreams or said prefetch horizon comprises monitoring periodicallysystem processing conditions including one or more of a number of readsand writes, the central processing unit (CPU) usage, and disk orinput/output (I/O) responses within a predetermined period of time, anddynamically varying to afford optimum system processing.
 6. The methodof claim 5, wherein said dynamically varying comprises varying thenumber of internal multiple streams accessing data from a singleaccessing stream to a number of multiple accessing streams to maintainsaid system processing conditions within predetermined limits.
 7. Themethod of claim 1, wherein said dynamically varying comprisesperiodically testing for compliance of said processing conditions andsaid preselected criteria with predetermined thresholds, and upon saidprocessing conditions or one or more of said preselected criteriafailing to satisfy said predetermined thresholds, backing off saidmultiple stream reading, and setting a retry time interval beforere-testing for compliance with said predetermined thresholds to renewmultiple stream reading.
 8. The method of claim 7, wherein said settinga retry time interval comprises setting a back-off factor, n, and aretry time unit, t, and setting said retry time as n*t, where saidback-off factor, n, is an integer that increments each time a retryfails to satisfy said predetermined thresholds, and the retry time unit,t, is a fixed unit of time.
 9. A non-transitory computer readablestorage medium embodying executable instructions for controlling aprocessor of a data storage system to perform a method of optimizingaccess time of sequential data in a file stored in said system inresponse to an external read request for a data restore, comprising:opening multiple internal read-ahead streams that read ahead in paralleldata in increments of a chunk of said file in said storage layer toprefetch said data; dynamically varying, to optimize processing andspeed of said data access, the number of said multiple read-aheadstreams that read said data or a prefetch horizon of the data read byeach of said multiple read-ahead streams based upon the present systemprocessing conditions and based upon a set of preselected criteriaapplicable to characteristics of the file being read; populating a cachewith the data read by said multiple read-ahead streams; and servicingsaid external read request with the data populated in the cache.
 10. Thenon-transitory computer readable storage medium of claim 9 furthercomprising confirming prior to opening said multiple read-ahead streamsthat the size of said file being accessed is greater than apredetermined threshold size below which the costs of setting up saidmultiple read-ahead stream access exceed the performance benefitobtained from multiple read-ahead stream access.
 11. The non-transitorycomputer readable storage medium of claim 9, wherein said preselectedcriteria include said file data being stored in a storage tier having aninput/output (I/O) bandwidth that is smaller than one at which theprocessing resource costs required for multiple stream reading exceedthe performance benefit obtained.
 12. The non-transitory computerreadable storage medium of claim 9, wherein said present systemprocessing conditions comprise a system processing load below apredetermined threshold load corresponding to said optimum processing.13. The non-transitory computer readable storage medium of claim 9,wherein said dynamically varying the number of read-ahead streams orsaid prefetch horizon comprises periodically monitoring systemprocessing conditions including one or more of a number of reads andwrites, the central processing unit (CPU) usage, and disk orinput/output (I/O) responses, and dynamically varying to afford optimumsystem processing.
 14. The non-transitory computer readable storagemedium of claim 13, wherein said dynamically varying comprises changingthe number of internal multiple streams accessing data from a singlestream to a number of multiple streams to maintain said systemprocessing conditions within predetermined limits.
 15. Thenon-transitory computer readable storage medium of claim 9, wherein saiddynamically varying comprises periodically testing for compliance ofsaid processing conditions and said preselected criteria withpredetermined thresholds, and upon said processing conditions or one ormore of said preselected criteria failing to satisfy said predeterminedthresholds, backing off said multiple stream reading, and setting aretry time interval before re-testing for compliance with saidpredetermined thresholds to renew multiple stream reading.
 16. Thenon-transitory computer readable storage medium of claim 15, whereinsaid setting a retry time interval comprises setting a back-off factor,n, and a retry time unit, t, and setting said retry time as n*t, wheresaid back-off factor, n, is an integer that increments each time a retryfails to satisfy said predetermined thresholds, and the retry time unit,t, is a fixed unit of time.